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1.  Introduction 


During  the  period  of  1  June  2006  through  01  December  2007,  the  Georgia  Tech  Research 
Institute  (GTRI)  engaged  in  the  project  "Facilitation  of  VSIPL++  Standardization,"  in  support  of 
the  High  Performance  Embedded  Computing  Software  Initiative  (HPEC-SI)  Program.  GTRI 
contributed  to  the  HPEC-SI  forum  objectives  by  assisting  in  the  development  of  Object  Oriented 
VSIPL  standards,  co-chairing  the  HPEC-SI  Development  Working  Group,  assisting  in  the 
dissemination  of  technical  designs  and  ideas,  and  the  maintenance  of  a  parallel  computing 
testbed  for  VSIPL++  experiments  by  HPEC-SI  program  participants. 

2.  Tasks  Completed 

GTRI  provided  specific  standards  prototypes  for  the  Working  Group  for  discussion  and 
implementation.  GTRI  served  on  the  Technical  Advisory  Board  of  the  HPEC-SI  program.  GTRI 
served  as  co-chair  for  the  HPEC-SI  Development  Working  Group.  GTRI  maintained  the 
website2  for  the  HPEC-SI  effort,  which  served  as  a  collection  point  for  information  about  the 
effort,  as  well  as  information  presented  at  the  meetings,  for  the  forum  members.  GTRI 
maintained  and  made  available  for  program  participants  a  parallel  computing  testbed. 

GTRI  attended  HPEC-SI  working  group  meetings  in  August  2006,  January  2007,  and  June  2007 
in  support  of  program  efforts.  At  these  meetings,  GTRI  participated  in  discussions  vetting  the 
proposed  VSIPL++,  parallel  VSIPL++,  and  proposals  to  expand  the  parallel  VSIPL++ 
specification  to  support  heterogeneous  and  tiled  computing  architectures. 

GTRI  continued  development  and  maintenance  of  the  HPEC-SI  Program  website,  including 
meeting  notes  and  presentations  for  each  of  the  Working  Group  meetings  during  the  project 
period,  conference  presentations,  and  prototype  applications.  In  conjunction  with  this,  GTRI 
continued  to  maintain  email  reflectors  for  use  by  HPEC-SI  program  participants. 

GTRI  made  a  parallel  computing  cluster  available  for  use  by  HPEC-SI  program3  participants  as 
a  testbed  for  VSIPL++,  parallel  VSIPL++,  and  other  parallel  computing  systems.  The  cluster  is  a 
fifty  five  node  Beowulf  style  cluster  with  116  compute  processors  of  varying  types.  Four  of  the 
compute  nodes  are  based  on  the  IBM/Toshiba/Sony  Cell  Broadband  Engine,  which  is  a 
platform  of  focus  for  the  HPEC-SI  program.  Several  HPEC-SI  program  participants  were  given 
or  maintained  login  access  during  the  project  and  used  the  testbed.  Software  support,  including 
the  installation  of  software  and  tools  useful  to  the  HPEC-SI  program  was  continued. 

GTRI  continued  to  maintain  the  VSIPL++  User's  Guide4.  The  Guide  is  intended  to  complement 
the  VSIPL++  Specification  Document  and  serve  as  an  introduction  and  clarification  of  the 
Specification  for  application  programmers.  The  guide  contains  elaborations  and  examples  from 
portions  of  the  VSIPL++  Specification  that  the  HPEC-SI  Working  Groups  find  to  be  difficult  or 
confusing  for  new  VSIPL++  application  programmers.  The  choices  of  topics  draw  heavily  from 
the  experiences  of  the  projects  undertaken  by  the  Demonstration  Working  Group,  as  well  as  the 
vetting  and  clarification  of  the  VSIPL++  Specification  by  the  Development  Working  Group. 

GTRI  continued  its  advisory  role  on  the  Technical  Advisory  Board  of  the  HPEC-SI  program,  as 
well  as  serving  as  the  co-chair  of  the  Development  Working  Group. 
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3.  Results 

3.1  Heterogeneous  &  Tiled  Parallel  extensions  to  VSIPL++ 

GTRI  participated  in  discussions  leading  up  to  the  conceptual  design  of  extensions  to  the 
Parallel  VSIPL++  specification5  designed  to  support  heterogeneous  computing  accelerators,  as 
well  as  tiled  chip  multiprocessors.  Examples  of  targeted  tiled  architectures  include  those 
developed  under  the  DARPA  Polymorphous  Computing  Architectures  program,  the 
IBM/ Sony/ Toshiba  Cell  Broadband  Engine,  and  commodity  multicore  general  purpose 
processors.  GTRI  was  not  the  primary  author  of  these  extensions,  but  nevertheless  was  actively 
involved  in  the  preliminary  discussions  in  anticipation  of  including  such  functionality  in  future 
versions  of  the  VSIPL++  specification. 

Extensions  to  Parallel  VSIPL++  primarily  focused  on  the  addition  of  task  and  communication 
abstractions.  These  abstractions  decouple  the  execution  of  portions  of  the  application  (tasks) 
and  define  an  interface  mechanism  (conduits)  between  the  tasks.  This  decoupling  allows 
execution  independence  among  the  tasks,  which  in  turn  allows  late  binding  of  tasks  to  actual 
compute  hardware,  more  efficient  utilization  of  each  hardware  element,  and  a  productive, 
abstract  method  of  defining  data  dependencies  between  tasks  at  the  application  level. 

3.2  Cluster-friendly  Playstation3  Unattended  Software  Build  Process 

GTRI  inserted  several  STI  Cell  Broadband  Engine  based  compute  nodes,  provided  by  the 
government,  into  the  parallel  software  testbed  for  use  by  E1PEC-SI  program  participants.  These 
compute  nodes  took  the  form  of  Sony  Playstation3  (PS3)  game  consoles.  Each  console  contains 
a  single  Cell  processor,  with  7  functioning  Synergistic  Processing  Elements  (SPEs).  The  Sony- 
provided  firmware  for  the  PS3  which  allows  users  to  install  third  party  operating  systems  that 
have  access  to  6  of  the  7  SPEs,  as  well  as  the  general  purpose  portion  of  the  Cell  processor.  The 
Linux  kernel  has  been  ported  to  support  the  Cell  processor,  and  many  distributions  of  the  Linux 
operating  system  have  been  ported  to  easily  install  on,  and  fully  exploit  the  Cell  processor  as 
available  from  a  third  party  operating  system.  In  order  to  support  the  goals  of  the  EIPEC-SI 
program,  the  government  supplied  four  PS3  consoles,  which  GTRI  inserted  into  to  the  parallel 
software  testbed. 

A  computer  that  is  used  as  a  compute  node  in  a  Beowulf-style  cluster  requires  a  means  of 
automated,  unattended,  definable  wipe  and  reinstall  of  system  software.  This  is  required  in 
order  to  guarantee  consistency  between  compute  nodes,  to  facilitate  system-wide  configuration 
changes,  and  to  allow  efficient  maintenance  and  troubleshooting  of  the  nodes.  This  capability  is 
widely  available  for  computers  that  are  frequently  used  in  such  a  manner,  but  is  not  for  the  PS3. 
All  of  the  widely  available  operating  system  installation  packages  require  human  intervention 
and  selection  of  configuration  options. 

In  order  to  allow  cost  effective  administration  of  the  PS3  compute  nodes,  as  well  as  facilitate 
future  upgrades  and  software  installation,  GTRI  developed  an  unattended,  automatic, 
configurable  system  wipe  and  reinstall  process  for  the  PS3.  A  cursory  description  of  this 
process  is  currently  available  electronically  (Reference  6). 
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The  specific  process  that  GTRI  developed  is  customized  to  the  Parallel  Software  Test  and 
Evaluation  Center  (PaSTEC)  testbed,  but  is  extendable  to  other  environments.  An  overview  of 
the  technique  used  is  described  in  the  following  paragraphs. 

A  modified  operating  system  boot  loader  (otheros.bld)  for  the  PS3  is  built  and  installed.  This 
boot  loader  will  query  a  specified  Trivial  File  Transfer  Protocol  (TFTP)  server  for  additional 
Linux  kernel  options  that  are  specified  by  the  system  administrator  for  the  network  address  of 
the  PS3  being  booted.  If  these  instructions  are  not  present,  the  PS3  will  boot  normally.  If 
present,  the  additional  kernel  options  can  be  set  to  instruct  the  PS3  to  load  and  execute  the 
Kickstart  process.  Kickstart  is  an  automated  version  of  the  Anaconda  installer  for  Red  Flat 
derived  Linux  distributions.  In  order  to  maximize  compatibility  with  Cell  processor  tools  and 
Cell  VSIPL++  development,  GTRI  elected  to  install  the  Fedora  Core  7  (a  Red  Hat  derived 
distribution)  operating  system  on  the  PS3s.  The  Kickstart  program  then  loads  further 
configuration  details  over  the  cluster  local  network  from  the  main  server,  formats  the  node-local 
hard  drive,  and  builds  the  distribution  on  the  PS3.  This  process  allows  system  administrators  to 
set  a  common  configuration  file  for  all  nodes  of  a  given  type,  and  to  directly  initiate  the  rebuild 
of  any  node,  from  software  controls. 

Several  elements  of  the  Kickstart  program  were  incomplete  or  faulty  for  the  Fedora  Core  7 
distribution  when  used  in  this  manner.  GTRI  found  and  enabled  a  workaround  for  each  of 
these  problems.  A  more  detailed  description  of  the  steps  required  is  available  via  the  World 
Wide  Web7.  This  description  was  written  as  a  high  level,  cursory  guide  for  system 
administrators,  a  capture  of  this  guide  as  of  February  20,  2008  is  shown  in  Figures  1  through  4. 
This  process  has  allowed  GTRI  to  deploy  the  PS3  nodes  into  the  parallel  software  testbed 
rapidly,  and  maintain  consistent  configuration  across  all  of  the  PS3  nodes,  through  several 
software  upgrades.  Using  this  process,  the  initial  deployment  of  the  four  PS3  nodes  into  the 
PaSTEC  cluster  required  approximately  30  minutes  of  operator  time  and  90  minutes  of 
unattended  time  to  complete. 
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PS3  Auto  Install  Process 

From  Cluster  Administration  Wild 


Contents 

■  1  Concept  of  Operations 

■  2  Prerequisites 

■  3  Build  customized  kboot 

■  4  Configure  DHCP  Server 

■  5  Configure  tftp  server 

■  6  Provide  kick  start  config  file 

■  7  Modify  and  serve  FC7  media 

■  8  Customize  kickstart  config  file 


Concept  of  Operations 

■  Tlus  process  is  modelled  loosely  on  the  PXE  Netboot  and  Anaconda  Kickstart  automated  mstall  process  widely  used 
for  ''Normal''  Fedora  Core  servers.  The  PS3  does  not  have  quite  the  same  netboot  capabilities  as  a  traditional  cluster 
server,  so  we  must  engage  in  some  tricks  to  mimic  the  behavior 

■  Compile  a  customized  kboot.  which  provides  a  new  otheros.bld  to  be  loaded  onto  the  PS3 

■  Configure  a  DHCP  server  to  provide  specific  addresses  to  known  PS3  nodes 

•  Early  in  the  boot  process,  after  obtaining  an  IP  address  via  DHCP,  new  kboot  asks  a  site-specific  tftp  server  if  an 
alternate  kboot.conf  has  been  assigned  to  that  IP  address 

■  If  not,  boot  will  proceed  normally,  skip  the  rest 

■  Special  kboot.conf  causes  an  install  kernel  and  lnitrdimg  to  be  downloaded  via  TFTP  and  sets  the  default  launch 
option  to  a  configuration  that  uses  the  install  images,  and  points  to  a  kickstart  configuration  file 

■  kickstart  configuration  file  is  obtained  via  NFS,  contains  anaconda  build  scnpt,  and  points  to  FC7  media  via  HTTP 

■  See  ours  (https:  pastec.gtn.gatech.edu  ks-fc7-ps3.cfg)  for  a  sample 

■  FC7  media  must  be  slightly  modified  to  work  with  YDL  installer.  FC7  installer  hangs  at  reboot. 

■  Kickstart  scnpt  is  fairly  typical,  with  some  exceptions: 

«  Mounts  mstall  from  front  node  via  NFS  which  contains  vanous  installation  files 

■  Installs  all  rpms  found  m  /install  ps3-fc7-rpms  directory  (including  an  upgraded  kernel  -  default  FC7  kernel 
hangs  on  reboot  attempt) 

■  Copies  SDK  3  .0  installation  files  to  local  drive,  and  creates  a  link  to  installtion  script  in  etc  rc.d 

■  Copies  a  bimch  of  cluster-wide  common  files,  including  etc  passwd.  /etc,  fstab,  etc  rhosts.  and  the  like 


Prerequisites 

■  PS3  already  mnning  hnux.  do  one  manual  install 

■  DHCP  server,  configured  to  deliver  static  IP  addresses  to  the  PS3  nodes 

■  Server  running  http,  nfs,  and  rsh/rcp  servers  that  the  node  to  be  installed  has  access  to. 

Build  customized  kboot 

■  This  step  will  build  a  custom  kboot  (otheros.bld)  that  will  check  a  tftp  server  for  alternate  boot  instructions 


https:  pastec.gtri.gatech.edu  admin-wiki  index.php/PS3_Auto_Install_Process 
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Figure  1.  PS3  Auto  Build  Guide  Page  1 
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■  download  modified  kboot  source  here  (http:  ftp-riken.go.jp  pub  Linux  kernel  people  eeoff  cell  ps3-lmux-distro- 
kit-200<S1208srckboot-20061208.tar.gz) 

■  Unpack  onto  ps3  already  running  YDL  or  FC 

■  Ensure  that  you  have  all  the  pre-requisites  required  to  compile  kboot. 

■  from  the  kboot- 10  directory  run  "make  download” 

■  from  the  top  level  kboot  directory,  ran  "make". 

■  when  the  make  is  complete,  create  a  special  kboot-10  root  etc.kboot.conf  to  ask  tftp  server  for  a  node-specific 
kboot.conf  on  boot,  if  available  Here  is  the  one  I  use 


root-LABIL-/ 

lpaiar-Vabln/ifconllg  ethO  |  grsp  *ln at  •  |  awx  '  (split  {$2,  all,  *  :  *  I ;  split  laid  [2]  ,  aldr,  • .  •)  ;  print  C  I  *  1 02X1 02X1 02X1 
02X\n*  ,ai51r  [1] ,  aidrC21  .  a<31ri3]  .  aditr[4] ) )' “ 

tftp  -1  /too  - r  pxaimux. rtg/Sipaldr  -g  130.207.222.09  2»  /dav/oull 

laf  aul t -yd 1 toxt 
cina:uc-5 
rcot-/  dar/adal 

yll- ' / dar/adal : /VTLI lnux -2 .6 . 16 •20061110 .ydl . 2ps3  lDltnl-/dfiv/edal : /lnltrd-2 . 6 . 16-20061110  .ydl .  2p b3 .  lmg  root-/ dar/sda2 
lnit-/ stln/lnlt  vlds:-ps3  fb:tr»ods  :3  rbgt ' 

yiliBOl-' /iav/slal  :  /rnl  imuc-2. 6 .16-20061110. ydl  .2pa3  Uiltrl-Zdar/adal:  /tnltrd-2 .6. 16-20061110. yll.  2p*3 .  lmg  ro=t-/lav/s-la2 
lnlt-/ at  ln/lnlt  vldar-p-a 3  fb  :trcds :  1  rbgt ' 

ylllOBOl- ' /dsv/cdal : /vmllnux-2 .6 .16-20061 110. ydl .2pa3  micid-/dav/adal : /lnltrd-2 .6.16-20061110 .ydl .2ps3 . lng  root-/dav/cda2 
lnlt-/vblQ/lnlt  vlda:«p-a3  fbtnods  :4  rbgt' 

ylltaxt-'  /dsv/odal  :/vmllnux-2 .6. 16-20061110.yll .  2p-a3  mitrd-/dav/clal :  /lnltrd-2. 6 . 16-20061110. ydl . 2pn 3  . lng  rcot-/dav/nda2 
mit-/«bin/init  3' 

aval  “cat  /too' 


■  ■  Note  on  above :  I  believe  the  ydl  lines  are  unnecessary'  but  have  not  tested.  My  installation  process  installs  a 

new  /etc kboot.conf  on  the  node  after  build,  and  this  is  used  if  the  tftp  fails. 

■  modify  the  tftp  line  as  necessary'  to  reach  your  local  server. 

■  rim  make  again 

■  from  the  kboot  top  level  directory,  execute: 


grip  -0  <  kt-=ot-lO/limix  >  tenants. bid 


■  Copy'  otheros.bld  to  USB  drive  and  install  as  per  the  usual  PS3  installation  process. 

Configure  DHCP  Server 

■  Obtain  the  MAC  address  of  each  PS3  to  be  installed 

■  Configure  the  DHCP  server  to  delver  a  static  IP  address  to  the  PS3 

Configure  tftp  server 

■  Alternative  kboot.conf  will  be  requested  from  the  compute  node 

■  The  boot  loader  will  look  for  a  file  that  has  the  name  of  the  hex  representation  of  the  node  s  IP  address 

■  e  g.  "0AFFFFC7"  is  the  file  for  10.253.255.199 

■  We  have  a  set  of  kboot  files  for  the  different  node  types  on  our  cluster  and  a  script  (https:// 
pastec.gtn.gatech.edu  rebuild)  that  creates  a  soft  link  to  the  appropnate  filename 

■  The  last  step  m  the  kickstart  process  removes  this  file  from  the  server,  so  that  subsequent  reboots  do  not  trigger 
an  install 

■  If  present,  this  file  takes  the  place  of  the  node's  default  kboot  .conf 

■  Below  is  the  file  used  on  this  system 
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|d«lauXC-ln*tallatlon 

lnscallatlco- •  //kernel  ks-nfc:  130. 207.222 .59 : /lnstall/klrlcBtart/ka-f c7-ps3  .cfg  lniex,d-//lnltra.gx  riiao-4S0l  inic-/ctm/ 
late  3' 

|:uei-'tctp  -9  -1  /lnitrd.gx  -r  lnierd-ycil-pa3.ing.gx  130. 207. 222. 59  »  /dar/null* 

|:uel-'eftp  -g  -1  /Jcarnal  -r  mllnuz-yll-pal  130.207.222.95  >  /dsv/null' 

Kmc  at  out  l 
kmc  st  foo 


•  m  note  the  method  for  obtaining  the  kickstart  file  (nfs).  We  found  that  for  the  YDL5  installer,  obtaining  the 
kickstart  configuration  file  via  http  led  to  a  crash 
■  place  the  specified  boot  files  in  the  tftp  director)' 

■  In  the  above  example,  we  use  the  imtrd  and  vmlinuz  that  come  from  the  Yellow  Dog  Linux  5  installation  boot 
media 


Provide  kickstart  coufig  file 

■  See  modifications  below 

■  place  a  kickstart  configuration  file  in  the  location  specified  by  the  "ks="  clause  in  the  kboot.conf  file 

■  For  us  tins  is  on  NFS  share  install  in  a  kickstart  director)'  with  a  specific  kickstart  config  file  for  each  node 
type 

■  You  can  see  our  directory  structure  [here  (https:  -pastec.etn  gatech.edu  install )  ]  -  it  is  served  via  NFS  as  well 
as  HTTP  HTTPS 

■  A  good  way  to  obtain  a  baseline  kickstart  config  file  is  to  do  a  manual  install,  and  then  grab  die  file  "  root  anaconda- 
ks.cfg" 

■  Several  modifications  are  required  to  integrate  mto  the  cluster  environment,  and  to  install  the  Cell  SDK.  see 
below 


Modify  and  serve  FC7  media 

■  The  YDL5  installer  can  not  install  from  the  raw  FC?  media,  but  slight  modifications  allow  it  work 

■  Download  and  unpack  a  FC7  install  image  into  a  directory 

■  from  FC7  root  directory. mkdir  YellowDog" 

■  "cd  YellowDog" 

■  "In  -s  ..-Fedora  RPMS" 

■  "mkdir  base:  cd  base" 

■  copy  "nnnstgLimg  from  a  YDL5  install  disk  "YellowDog base.  mmstg2.img" 

■  "cp  ..  ..  images  stagelimg  ." 

■  link  or  copy  the  FC7  directory  to  the  web  server  location  specified  in  the  kickstart  config  file 

Customize  kickstart  config  file 

■  Here  (https:-7pastec.gtn  gatech.edu  install kickstartks-fc7-ps3  cfg)  is  ours 

■  Customize  package  list  to  fit  sour  needs 

■  Remember  to  set  the  encrypted  root  password  as  desired.  We  have  one  duster-wide  root  password 

■  Notes  on  kickstart  configuration: 

■  must  "mkdir  spu"  so  that  the  spufs  can  be  mounted 

■  we  install  kernel  2,6.23-rc3.  because  the  FC7  stock  kernel  hangs  on  reboot 

■  our  cluster  has  a  NFS  Share  on  the  front  node  at  install.  Several  steps  in  our  kickstart  config  reference  files 
there 

■  We  were  not  able  to  make  installation  of  the  Cell  SDK  work  _dunng_  the  OS  install,  so  we  use  a  hack  to  cause 
it  to  be  installed  the  first  subsequent  time  the  system  is  booted: 
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cp  /*mt/tfinp/tlles/pa3-£c7-c«ll»3X-Uistall  /etc/rc.d/rc.  oycinit.  aueiior 

Lrb;  *1C  I  - 1  /ate/ rc.d/rc .cysinic. author  ]  ;  cjiqd  /atc/rc  .d/rc .cya  ln.it. autbsr;  Cl*  »  /atc/rc .d/rc .local 
clmod  7SS  /atc/rc  .d/rc  .ayalntt  .author 


■  ■  ■  Here  (https:  pastec.gm.gatech.edu  ps3-fc?-cellsdk-install)  is  die  ps3-fc7-cellsdk -install  script.  This  script 

runs  the  cellsdk  process.  The  hrtp_proxy  variables  are  set  to  allow  the  node  to  access  the  outside  world 
dining  the  yum  steps.  Without  this,  the  process  takes  hours  to  complete  for  us.  We  run  a  htp  proxy  on  port 
3148  of  the  front  node. 

■  Install  any  rpms  in  the  nfs:  install  rpmsps3-fc7-rpm  directory'.  We  put  any-  extra  iprns  that  need  to  be  installed 
for  this  platform  in  that  location 

■  Copy  over  the  system  kboot.conf  file  from  the  template  on  the  file  server 

■  Copy  over  the  local  yum  repository'  to  the  node 

■  Call  the  ks-include  script 

■  This  script  contains  the  common,  system-wide  file  copying  done  by  all  nodes  of  all  types 

■  nfs  mount  points,  shadow  and  passyvord  files,  group,  hosts,  fstab,  ssh  and  rsh  config  files 

■  remove  the  ip-specific  install  trigger  file  from  the  front  node’s  tftp  directory' 

■  Here  (https:pastec.gtn  gatech.edu  install  kickstart/ks-inchide)  is  ours. 

■  install  gmond 

■  turn  on  and  off  a  few  services 

Retrieved  from  "https:  pastec.gtri.gatech.edu  admm-wiki  mdex  php  PS3  Auto  Install  Process" _ 

■  This  page  was  last  modified  19:07, 29  December  2007. 
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4.  Conclusions 


During  the  period  of  performance  of  the  subject  contract,  the  Georgia  Tech  Research  Institute 
contributed  to  the  High  Performance  Embedded  Computing  Software  Initiative  program  as  a 
technical  participant,  and  as  a  member  of  the  technical  advisory  board.  GTRI  developed 
software  strategies,  assisted  in  the  dissemination  of  program  results  via  internet  tools,  provided 
a  parallel  software  testbed  for  program  participants,  and  developed  a  novel  method  for  the 
unattended  deployment  of  Linux-based  operating  systems  onto  PS3  compute  nodes  in  a 
Beowulf-style  cluster.  GTRI  also  participated  in  technical  advisory  planning  for  the  HPEC-SI 
program. 
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List  of  Acronyms 


API 

Application  Programming  Interface 

DARPA 

Defense  Advanced  Research  Projects  Agency 

GTRI 

Georgia  Tech  Research  Institute 

HPEC-SI 

High  Performance  Embedded  Computing  Software  Initiative 

PCA 

Polymorphous  Computing  Architectures 

PS3 

Playstation3 

SPE 

Synergistic  Processing  Element 

TFTP 

Trivial  File  Transfer  Protocol 

VSIPL 

Vector  Signal  &  Image  Processing  Library 

9 


